Integral Reinforcement Learning Research Articles

Intelligent adaptive lighting algorithm: Integrating reinforcement learning and fuzzy logic for personalized interior lighting

The lighting requirements are subjective and one light setting cannot work for all. However, there is little work on developing smart lighting algorithms that can adapt to user preferences. To address this gap, this paper uses fuzzy logic and reinforcement learning to develop an adaptive lighting algorithm. In particular, we develop a baseline fuzzy inference system (FIS) using the domain knowledge, generating light recommendation based on a set of intuitive rules. These rules, derived from existing literature, are based on environmental conditions i.e. daily glare index, and user information including age, activity, and chronotype. Through a feedback mechanism, the user interacts with the algorithm, correcting the algorithm output to their preferences. We interpret these corrections as rewards to a Q-learning algorithm, which tunes the FIS parameters online to match the user preferences. The Q-learning is a model-free learning algorithm that learns to act optimally by interacting with the user and the rewards it receives. This allows the proposed algorithm to work in a model-free manner, effectively handling the uncertainties that might arise from the individualistic preferences of users. To the authors’ best knowledge, this algorithm is pioneering work in designing intelligent algorithms for personalized lighting control, featuring several elements of novelty, including the number of environmental and user-related inputs, the continuous control of light intensity as opposed to common on/off control, and the ability to learn user preferences. The algorithm is implemented in a real aircraft cabin and is evaluated in an extensive user study. The implementation results demonstrate that the developed algorithm possesses the capability to learn user preferences while successfully adapting to a wide range of environmental conditions and user characteristics. This underscores its viability as a potent solution for intelligent light management, featuring advanced learning capabilities.

Adaptive Critic Learning-Based Optimal Bipartite Consensus for Multiagent Systems With Prescribed Performance.

Developing a distributed bipartite optimal consensus scheme while ensuring user-predefined performance is essential in practical applications. Existing approaches to this problem typically require a complex controller structure due to adopting an identifier-actor-critic framework and prescribed performance cannot be guaranteed. In this work, an adaptive critic learning (ACL)-based optimal bipartite consensus scheme is developed to bridge the gap. A newly designed error scaling function, which defines the user-predefined settling time and steady accuracy without relying on the initial conditions, is then integrated into a cost function. The backstepping framework combines the ACL and integral reinforcement learning (IRL) algorithm to develop the adaptive optimal bipartite consensus scheme, which contributes a critic-only controller structure by removing the identifier and actor networks in the existing methods. The adaptive law of the critic network is derived by the gradient descent algorithm and experience replay to minimize the IRL-based residual error. It is shown that a compute-saving learning mechanism can achieve the optimal consensus, and the error variables of the closed-loop system are uniformly ultimately bounded (UUB). Besides, in any bounded initial condition, the evolution of bipartite consensus is limited to a user-prescribed boundary under bounded initial conditions. The illustrative simulation results validate the efficacy of the approach.

Safe tracking in games: Achieving optimal control with unknown dynamics and constraints

AbstractThis paper introduces mix‐zero‐sum differential (MZSD) game theory to address multi‐player tracking systems, offering a better understanding of the coexistence of cooperation and competition among players. Within this framework, we present an optimal safety tracking control (OSTC) method, which incorporates a control barrier function (CBF) into the value function to ensure that the tracking error remains within a specified range, thus guaranteeing safety while achieving optimization. Simultaneously, to eliminate the need for system dynamics, we propose a novel approach leveraging off‐policy integral reinforcement learning (IRL) technology to obtain the Nash equilibrium solution of the MZSD games. We establish a unique critics–actors neural network (NN) structure that updates concurrently. Furthermore, we analyze stability and convergence using the Lyapunov method. We conduct two simulations to demonstrate the effectiveness of the proposed algorithm.

Integrating Reinforcement Learning and Metaheuristics for Safe and Sustainable Health Tourist Trip Design Problem

This study aims to address the critical challenge of enhancing safety within the domain of health and wellness tourism through the framework of sustainable travel design. Given the rising demand for secure and sustainable tourism experiences, this investigation explores the integration of safety considerations into the strategic planning and execution of tourist journeys. The central focus of this research revolves around the development and application of the Modified Reinforcement Learning-Artificial Multiple Intelligence System (MRL-AMIS). Operating within a multi-objective framework, the objective of this study is to optimize essential parameters encompassing environmental sustainability, economic viability, tourist preferences, and societal acceptability, with a specific emphasis on safety. The efficacy of the proposed methodology is rigorously evaluated across various tourist group scenarios, providing a comprehensive assessment of its applicability. Our computational analyses unequivocally demonstrate the superiority of the proposed approach in enhancing safety, surpassing existing methods such as genetic algorithms, differential evolution algorithms, particle swarm optimization, etc., by 15%-25%. These findings underscore the effectiveness of the proposed method in seamlessly integrating safety considerations into the realm of sustainable tourism planning. This study highlights the potential of the proposed method as an advanced tool in the strategic planning of sustainable tourism, particularly within the context of health and wellness tourism. It constitutes a significant contribution to the field of sustainable tourism by offering an innovative approach that prioritizes safety while upholding the principles of environmental stewardship, economic viability, and societal sustainability.

Drug Dosage Control System Using Reinforcement Learning

This project introduces a pioneering approach for optimizing drug dosage control strategies through the utilization of reinforcement learning (RL), a sophisticated subset of machine learning techniques. The core objective is to dynamically adjust drug dosages in real-time based on patient responses, thereby maximizing therapeutic efficacy while minimizing potential adverse effects. By integrating reinforcement learning algorithms, including Q-learning, Deep Q-Networks (DQN), and actor-critic methods, the system learns from patient data to make precise dosage adjustments considering individual patient characteristics, disease progression, and response to treatment. The framework promises to revolutionize personalized medicine by providing tailored drug dosages, enhancing treatment outcomes, and ensuring patient safety. The project's scope covers not only the development and implementation of this innovative RL- based system but also addresses significant challenges such as model interpretability, scalability, and regulatory compliance, ensuring its practical applicability in healthcare settings. Through this work, we aim to bridge the gap between conventional drug prescription methodologies and the potential for personalized, optimized care, making a substantial contribution to the advancement of healthcare systems.

Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning

AbstractThis paper studies the cooperative tracking control problem of interacted multi‐agent systems (MASs) under undirected communication. Based on differential graphical game theory, the MAS tracking control problem is formulated as an infinite horizon cooperative differential graphical game‐theoretic tracking control framework, where a multi‐objective optimization problem is designed and then cast into a Pareto‐equivalent single‐objective optimization problem using a scalarization method. Necessary and sufficient conditions for the existence of the Pareto‐optimal strategy to the game theoretic tracking control are established, where it has been proven that the solution to the integral Bellman optimality equation leads to Pareto‐optimal strategy. Then, an off‐policy integral reinforcement learning scheme to find optimal control strategy using a pure data‐driven manner is developed, which consumes less computation efforts than the traditional learning scheme. Simulated results are conducted to validate the effectiveness of the proposed game and IRL‐based tracking control method.

Integrating reinforcement learning and serious games to support people with rare genetic diseases and neurodevelopmental disorders: outcomes on parents and caregivers.

Integrating reinforcement learning and serious games to support people with rare genetic diseases and neurodevelopmental disorders: outcomes on parents and caregivers.

Integral reinforcement learning-based dynamic event-triggered safety control for multiplayer Stackelberg–Nash games with time-varying state constraints

In this article, an integral reinforcement learning-based dynamic event-triggered safety control scheme is proposed to tackle the multiplayer Stackelberg–Nash games (MSNG) problem in continuous-time nonlinear systems with time-varying state constraints. Initially, a new barrier function (BF) is introduced by integrating traditional barrier functions with a novel smooth function to address the challenge of time-varying state constraints. Then, the constrained MSNG system is transformed into an unconstrained system through the application of the state transformation technique, which helps to characterize hierarchical decision problems as MSNG problems with the leader and followers. Meanwhile, the integral reinforcement learning (IRL) technique is also applied to ease the demand for precise system dynamics. Moreover, a new dynamic event-triggered control (DETC) mechanism is designed, resulting in coupled dynamic event-triggered Hamilton–Jacobi (HJ) equations. A single critic neural network (NN) is constructed to learn the optimal control laws for the leader and followers. By employing Lyapunov theory, the integral reinforcement learning-based dynamic event-triggered safety control method ensures that the system state and the critic NN weight errors are uniformly ultimately bounded (UUB). Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed algorithm.

Integral reinforcement learning-based angular acceleration autopilot for high dynamic flight vehicles

During the synthesis of acceleration autopilots for high dynamic flight vehicles (HDFV), autopilots with feedback of angular acceleration (AFAA) have become more perspective with stringent requirements on response speed and high maneuverability, compared with autopilots with feedback of angular rate (AFAR). Integral reinforcement learning (IRL) method has now proved to be an effective technique for adaptive optimal control of partially unknown nonlinear systems. In this paper, a novel data-driven IRL algorithm with “actor-critic” structure is proposed for HDFV utilizing AFAA. As an advanced model-free approach, actor-critic based IRL algorithm learns optimal behaviors by observing the real-time responses from the environment under the action of nonoptimal control policies. Instead of solving algebraic Riccati equation directly, the control policy updates online via the solution of proposed IRL Bellman equation with sensed quantities. Numerical simulation is carried out to validate the effectiveness of proposed online IRL-based angular acceleration autopilot for HDFVs. Besides, the tracking performance under different wave commands, the robustness against parameter uncertainties and the noise attenuation capacity between classical optimal tracking approach and proposed IRL method are analyzed for AFAR and AFAA, respectively. Simulation results show that, angular acceleration autopilot with proposed integral RL algorithm possesses better tracking performance against various disturbances.

STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration

Experience-based methods like reinforcement learning (RL) are often deemed less suitable for the safety field due to concerns about potential safety issues. To bridge this gap, we introduce STPA-RL, a methodology that integrates RL with System-Theoretic Process Analysis (STPA). STPA is a safety analysis technique that identifies causative factors leading to unsafe control actions and system hazards through loss scenarios. In the context of STPA-RL, we formalize the Markov Decision Process based on STPA analysis results to incorporate control algorithms into the system environment. The agent learns safe actions through reward-based learning, tracking potential hazard paths to validate system safety. Specifically, by analyzing various loss scenarios related to the Platform Screen Door, we assess the applicability of the proposed approach by evaluating hazard trajectory graphs and hazard frequencies in the system. This paper streamlines the RL process for loss scenario identification through STPA, contributing to self-guided loss scenarios and diverse system modeling. Additionally, it offers effective simulations for proactive development to enhance system safety and provide practical assistance in the safety field.

Design optimization of obstacle avoidance of intelligent building the steel bar by integrating reinforcement learning and BIM technology

In promoting the construction of prefabricated residential buildings in Yunnan villages and towns, the use of precast concrete elements is unstoppable. Due to the dense arrangement of steel bars at the joints of precast concrete elements, collisions are prone to occur, which can affect the stress of the components and even pose certain safety hazards for the entire construction project. Because the commonly used the steel bar obstacle avoidance method based on building information modeling has low adaptation rate and cannot change the trajectory of the steel bar to avoid collision, a multi-agent reinforcement learning-based model integrating building information modeling is proposed to solve the steel bar collision in reinforced concrete frame. The experimental results show that the probability of obstacle avoidance of the proposed model in three typical beam-column joints is 98.45%, 98.62% and 98.39% respectively, which is 5.16%, 12.81% and 17.50% higher than that of the building information modeling. In the collision-free path design of the same object, the research on the path design of different types of precast concrete elements takes about 3–4 minutes, which is far less than the time spent by experienced structural engineers on collision-free path modeling. The experimental results indicate that the model constructed by the research institute has good performance and has certain reference significance.

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Extensive research has been carried out on reinforcement learning methods. The core idea of reinforcement learning is to learn methods by means of trial and error, and it has been successfully applied to robotics, autonomous driving, gaming, healthcare, resource management, and other fields. However, when building reinforcement learning solutions at the edge, not only are there the challenges of data-hungry and insufficient computational resources but also there is the difficulty of a single reinforcement learning method to meet the requirements of the model in terms of efficiency, generalization, robustness, and so on. These solutions rely on expert knowledge for the design of edge-side integrated reinforcement learning methods, and they lack high-level system architecture design to support their wider generalization and application. Therefore, in this paper, instead of surveying reinforcement learning systems, we survey the most commonly used options for each part of the architecture from the point of view of integrated application. We present the characteristics of traditional reinforcement learning in several aspects and design a corresponding integration framework based on them. In this process, we show a complete primer on the design of reinforcement learning architectures while also demonstrating the flexibility of the various parts of the architecture to be adapted to the characteristics of different edge tasks. Overall, reinforcement learning has become an important tool in intelligent decision making, but it still faces many challenges in the practical application in edge computing. The aim of this paper is to provide researchers and practitioners with a new, integrated perspective to better understand and apply reinforcement learning in edge decision-making tasks.

Reinforcement learning based sliding mode control for two-time-scale multi-agent systems with malicious attacks

The optimal consensus problem for linear two-time-scale multi-agent systems under malicious attacks is studied in this paper. Firstly, an integral sliding mode function is devised to guide the system trajectory towards the sliding mode surface and the impact of attacks can be eliminated. Then, the optional consensus problem is reformulated as a zero-sum game problem between each agent and its neighbouring agents. Thus, the game algebraic Riccati equation with singularly perturbed parameter is formulated. Furthermore, to avoid the requirement of the system dynamics information, an integral reinforcement learning algorithm is presented to obtain the optimal control policy for multi-agent systems. Compared with existing learning methods, the obtained reinforcement learning algorithm is devoid of potential calculation error issues from singularly perturbed parameter. Meanwhile, the convergence of the proposed algorithm is verified. Finally, a simulation example is provided to demonstrate the efficacy of the proposed control method.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

Behavioral selection in structured populations.

The multilevel model of behavioral selection (MLBS) by Borgstede and Eggert (Behav Process 186:104370. 10.1016/j.beproc.2021.104370 , 2021) provides a formal framework that integrates reinforcement learning with natural selection using an extended Price equation. However, the MLBS is so far only formulated for homogeneous populations, thereby excluding all sources of variation between individuals. This limitation is of primary theoretical concern because any application of the MLBS to real data requires to account for variation between individuals. In this paper, I extend the MLBS to account for inter-individual variation by dividing the population into homogeneous sub-populations and including class-specific reproductive values as weighting factors for an individual's evolutionary fitness. The resulting formalism closes the gap between the theoretical underpinnings of behavioral selection and the application of the theory to empirical data, which naturally includes inter-individual variation. Furthermore, the extended MLBS is used to establish an explicit connection between the dynamics of learning and the maximization of individual fitness. These results expand the scope of the MLBS as a general theoretical framework for the quantitative analysis of learning and evolution.

Dynamic Load Balancing in Cloud Computing: Optimized RL-Based Clustering with Multi-Objective Optimized Task Scheduling

Dynamic load balancing in cloud computing is crucial for efficiently distributing workloads across available resources, ensuring optimal performance. This research introduces a novel dynamic load-balancing approach that leverages a deep learning model combining Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to calculate load values for each virtual machine (VM). The methodology aims to enhance cloud performance by optimizing task scheduling and stress distribution. The proposed model employs a dynamic clustering mechanism based on computed loads to categorize VMs into overloaded and underloaded clusters. To improve clustering efficiency, the approach integrates Reinforcement Learning (RL) with a sophisticated Hybrid Lyrebird Falcon Optimization (HLFO) algorithm. HLFO merges the Lyrebird Optimization Algorithm (LOA) and Falcon Optimization Algorithm (FOA), enhancing the effectiveness of load balancing. A Multi-Objective Hybrid Optimization model is introduced to optimize task scheduling while considering Quality of Service (QoS) parameters, including makespan minimization, energy consumption reduction, balanced CPU utilization, efficient memory usage, and task prioritization. The implementation, conducted in Python and CloudSim, demonstrates the model’s ability to effectively allocate work between virtual machines (VMs) and physical machines (PMs), resulting in improved resource utilization, shortened makespan, enhanced CPU usage, and rigorous assessments affirming its efficacy. This research addresses the complexity of dynamic load balancing in cloud environments by combining deep learning, reinforcement learning, and hybrid optimization techniques, offering a comprehensive solution to optimize cloud performance under varying workloads and resource conditions.

Two-player nonlinear Stackelberg differential game via off-policy integral reinforcement learning

For the nonzero-sum games, players have taken different strategies to achieve the Nash equilibrium. However, regarding hierarchical optimization and asymmetric information, Nash equilibrium is ineffective. The Stackelberg game provides us with a leader-follower strategy to cope with this problem. This paper solved two-player continuous-time nonlinear Stackelberg differential games with unknown system dynamics using an off-policy integral reinforcement learning (IRL) technique. A two-level hierarchy optimal control problem is the Stackelberg differential game, and locating the open-loop Stackelberg equilibrium is equivalent to solving the coupled Hamiltonian-Jacobi (HJ) equations. First, the optimal strategies for the leader and the follower are derived. Then, the closed-loop system’s asymptotic stability of optimal control solution is proved. Based on the policy iteration (PI) algorithm, an off-policy IRL algorithm is developed to solve the Stackelberg game for completely unknown system dynamics. The control strategies and cost functions of the leader and follower are approximated using neural networks (NNs). It is demonstrated that NNs weight errors of off-policy algorithm are uniformly ultimately bounded (UUB). Lastly, two simulation examples demonstrate that the proposed learning algorithm is capable of obtaining two-player Stackelberg equilibrium strategies.

Leader-Follower Based Online Reinforcement Learning Algorithm in Problem with Hierarchy Decision Makers

This paper presents an online integral reinforcement learning (RL) solution for problems with hierarchy decision makers. Specifically, we reformulate this model as a leader-follower game in which control input and deterministic disturbance act as decision makers at different levels of hierarchy: the control input plays the role of the leader while the disturbance plays the role of the follower. The main contributions of this paper can be summarized as follows. First, we introduce online RL to deal with systems that have partially unknown information, meaning that accurate dynamic information is not required. Second, we solve the leader-follower coupled Hamilton-Jacobi (HJ) and Riccati equations approximately online using the derived algorithm. Third, we provide turning laws for cost functions and controllers that ensure closed-loop stability simultaneously.

Reinforcement learning-assisted evolutionary algorithm: A survey and research opportunities

Evolutionary algorithms (EA), a class of stochastic search methods based on the principles of natural evolution, have received widespread acclaim for their exceptional performance in various real-world optimization problems. While researchers worldwide have proposed a wide variety of EAs, certain limitations remain, such as slow convergence speed and poor generalization capabilities. Consequently, numerous scholars actively explore improvements to algorithmic structures, operators, search patterns, etc., to enhance their optimization performance. Reinforcement learning (RL) integrated as a component in the EA framework has demonstrated superior performance in recent years. This paper presents a comprehensive survey on integrating reinforcement learning into the evolutionary algorithm, referred to as reinforcement learning-assisted evolutionary algorithm (RL-EA). We begin with the conceptual outlines of reinforcement learning and the evolutionary algorithm. We then provide a taxonomy of RL-EA. Subsequently, we discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. The RL-assisted procedure is divided according to the implemented functions including solution generation, learnable objective function, algorithm/operator/sub-population selection, parameter adaptation, and other strategies. Additionally, different attribute settings of RL in RL-EA are discussed. In the applications of RL-EA section, we also demonstrate the excellent performance of RL-EA on several benchmarks and a range of public datasets to facilitate a quick comparative study. Finally, we analyze potential directions for future research. This survey serves as a rich resource for researchers interested in RL-EA as it overviews the current state-of-the-art and highlights the associated challenges. By leveraging this survey, readers can swiftly gain insights into RL-EA to develop efficient algorithms, thereby fostering further advancements in this emerging field.

Application of integrating reinforcement learning and intelligent scheduling in logistics distribution

Logistics distribution is an indispensable part of the modern economy and is crucial for ensuring the efficient operation of the supply chain. With the continuous progress of technology, the application of intelligent scheduling systems in the field of logistics distribution is becoming increasingly widespread. Reinforcement learning, as one of the hot technologies in the field of artificial intelligence, is gradually receiving attention in its application in intelligent scheduling. Reinforcement learning can continuously learn and predict the same thing to enhance memory, while intelligent scheduling requires continuous prediction and optimization of logistics distribution paths. In response to the current problems of slow logistics distribution efficiency and low customer satisfaction, this article analyzed the application of intelligent scheduling in logistics distribution from the aspects of basic data maintenance, basic data review, intelligent scheduling, scheduling result review, distribution information management, and vehicle tracking. By using reinforcement learning, the traffic network weight in logistics distribution was studied to improve logistics distribution efficiency and customer satisfaction. This article analyzed the efficiency of logistics distribution, vehicle tracking accuracy, vehicle scheduling ability, and logistics distribution costs under different logistics distributions. The results showed that the logistics distribution under the integration of reinforcement learning and intelligent scheduling reduced 12.047 km compared to traditional distribution paths, and its distribution cost decreased by 129.718 yuan compared to traditional logistics distribution costs. The efficiency of logistics distribution that integrates reinforcement learning and intelligent scheduling has significantly improved, with optimized distribution costs and paths. It also has a positive effect on improving the utilization rate of logistics distribution vehicles.